This started as a simple refactor, but during this, I discovered a few issues too, which should now be fixed. This is probably best reviewed by checking each commit separately.

Here are the descriptions for the individual changes:

1. Refactor: Move LoRA update_layer to child classes

For LoRA, so far, we have update_layer for Linear, update_layer_embedding for Embedding, and update_layer_conv2d for Conv2d, all defined on LoraLayer.

We can simplify the code by always using the name update_layer, and by moving the layer-specific methods to the subclasses. So e.g. update_layer_embedding is moved to the Embedding class and renamed to update_layer. This way, the caller does not need to differentiate which type of layer it's calling, making it much easier (see the simplification in LoraModel._create_and_replace). It also makes the code less error prone (see change 4 described below).

Interestingly, this was already practiced for IA³, so the same change was not necessary there. But I did find the same method implemented twice, once on IA3Layer and once on Linear, so I removed one of the duplicates. For all other adapter methods, no change was required, as they only implement Linear.

Note: This could technically be backwards incompatible if users do some custom stuff with LoraLayers. I could add a note for next release that all update_layer_* methods have been consolidated to use the same update_layer name.

2. Systematic handling of r (rank) <= 0

Always raise an error when r <= 0, not only for LoRA. Also, removed later check for r > 0 in LoRA layers, since we already check for r <= 0.

Note: This could technically also be considered backwards incompatible, but r<=0 should not work correctly anyway, so better to raise an error right away.

3. Fix broken __repr__ method on QuantLinear

Was indented too deep, thus not being applied.

4. Fix bug for updating Lora GPTQ and IA3 bnb layers

Before this fix, when adding a 2nd adapter to a model, we did not correctly check if there was already an adapter layer in the model when dealing with LoRA GPTQ or IA3 bnb layers. As a consequence, instead of updating the existing layers, we would create a new layer and the existing layer would be set as the base_layer of that new layer. Now, we correctly update the existing layer to add the new adapter.

For this fix to work correctly with LoRA and GPTQ, I had to add a check for qweight in update_layer, since we only checked for weight before. This was a mistake that didn't surface so far because of the error described in the previous paragraph.

Tests were extended to check this. They fail with the current main but pass on my machine with this PR.